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SYSTEM AND METHOD OF NON-LINEAR GRID FITTING 
AND COORDINATE SYSTEM MAPPING 

CROSS-REFERENCE TO RELATED APPLICATIONS 

The present application claims the benefit of United States provisional application 
5 Serial No. 60/454,581, filed March 14, 2003, entitled "AN APPROACH FOR NONLINEAR 

GRID FITTING AND COORDINATE SYSTEM MAPPING," the disclosure of which is 
hereby incorporated herein by reference in its entirety. 

FIELD OF THE INVENTION 

Aspects of the present invention relate generally to coordinate system mapping 
10 applications, and more particularly to a system and method of non-linear grid fitting and 

coordinate system mapping for image acquisition and data processing applications. 

BACKGROUND OF THE INVENTION 

In many conventional image acquisition and image data processing systems, feature 
geometry of a known, highly accurate artifact may become distorted during imaging, data 

1 5 processing, or both. One such situation may arise where a precision Cartesian grid (or array) 

of points printed on glass or other substrate material is imaged using optics and a camera, 
such as a charge-coupled device (CCD) camera, for example, or a complementary metal- 
oxide semiconductor (CMOS) imaging device. Such artifact features may be referred to as 
fiducials, and the foregoing substrate having a known pattern of fiducials printed thereon, or 

20 incorporated into the structure thereof, may be referred to as a fiducial plate. 

FIG. 1 is a simplified diagram illustrating raw image data acquired by an imaging 
apparatus and representing a top view of a precision Cartesian grid of fiducials printed on a 
fiducial plate. Image acquisition of such fiducial plates and fiducial arrays may have utility 
in various contexts such as semiconductor probe card testing processes, for example, 

25 calibration of high-resolution imaging apparatus, and other imaging applications requiring a 

high degree of accuracy. 

Given a precision artifact such as a fiducial or a fiducial array or grid to be imaged, 
conventional technology is deficient to the extent that it lacks the ability to identify the non- 
linear transformation to and from imaged coordinates (i.e., coordinates derived from 

30 acquired image data) and artifact coordinates on the fiducial plate (i.e., actual coordinates of 

the fiducial relative to a reference point on the fiducial plate). 
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Specifically, in an acquired image image data obtained by a camera or other 
imaging hardware), the respective location of the center of each respective fiducial may be 
extracted from the acquired image iiata. In the case where the fiducial plate carries a 
Cartesian grid of fiducials, ideally, the fiducial locations in the acquired image form a 
5 regular, known rectangular grid aligned, for example, with the axes of the camera or other 

imaging apparatus. Due to factors such as fiducial absence, stage rotations, camera 
rotations, pixel size variation, magnification variation, keystone/barrel distortion, and other 
optical or mechanical effects, the measured fiducial locations may deviate from the ideal 
regular rectangular grid (e.g., as it exists on the Cartesian array of the fiducial plate). 

10 SUMMARY 

Aspects of the present invention overcome the foregoing and other shortcomings of 
conventional technology, providing a system and method of non-linear grid fitting and 
coordinate system mapping for image acquisition and data processing applications. 
Exemplary embodiments may model the non-linear transformation to and from imaged 
15 coordinates (i.e., coordinates derived from acquired image data) and artifact coordinates on 

the fiducial plate (i.e., actual coordinates of the fiducial relative to a reference point on the 
fiducial plate). 

In accordance with one exemplary embodiment, a method of fitting acquired fiducial 
data to a set of fiducials on a fiducial plate may comprise: fitting a fiducial grid model to 

20 data acquired by an imaging apparatus; establishing a conversion from acquired coordinates 

to ideal fiducial coordinates; and calculating an absolute location of identified acquired 
image feature centers in fiducial plate coordinates. As set forth in more detail below, the 
fitting operation may comprise identifying fiducial coordinates for each fiducial captured in 
the data acquired by the imaging apparatus. 

25 Additionally, some disclosed methods may further comprise selectively iterating the 

identifying coordinates for each fiducial and the calculating an absolute location of identified 
acquired image feature centers. 

In accordance with one embodiment of such a method, the calculating comprises 
utilizing a linear least squares operation. Additional exemplary embodiments may comprise 

30 assuming that a rotation of the imaging apparatus relative to a fiducial grid is negligible. 
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Embodiments are described wherein the imaging apparatus comprises a charge- 
coupled device camera, a complementary metal-oxide semiconductor device, or similar 
imaging hardware. 

In another exemplary embodiment, a method of accurately measuring a location of a 
5 feature relative to a known set of fiducials comprises: acquiring image data; responsive to 

the acquiring, representing a location of a fiducial in a local fiducial space coordinate 
system; and mapping a coordinate in the local fiducial space coordinate system to a 
corresponding location in an image apparatus space. As set forth in detail below, the 
mapping operation in some embodiments comprises employing a polynomial fit in terms of 

10 fiducial coordinates; such employing comprises utilizing a second order polynomial fit, a 

third order polynomial fit, or some other suitable function. 

In accordance with another aspect, a method of fitting a set of measured fiducial data 
to an ideal set of fiducials, where the fiducials are arranged in a Cartesian grid pattern on a 
substantially transparent substrate, comprises: acquiring the measured fiducial data 

15 employing an imaging apparatus; responsive to the acquiring, representing a location of a 

fiducial in a local fiducial space coordinate system; and mapping a coordinate in the local 
fiducial space coordinate system to a corresponding location in a space associated with the 
image apparatus. As with the method identified above, the mapping in some embodiments 
may comprise employing a polynomial fit in terms of fiducial coordinates. Such a 

20 polynomial fit may be second order, third order, or higher order, for example. 

In accordance with another aspect of the disclosed subject matter, a computer 
readable medium may be encoded with data and instructions for fitting acquired fiducial data 
to a set of fiducials on a fiducial plate; the data and instructions may cause an apparatus 
executing the instructions to: fit a fiducial grid model to data acquired by an imaging 

25 apparatus; establish a conversion from acquired coordinates of each identified fiducial to 

ideal fiducial coordinates; and calculate an absolute location of identified acquired image 
feature centers in fiducial plate coordinates. 

As set forth in more detail below, the computer readable medium may be further 
encoded with data and instructions for causing an apparatus executing the instructions to 

30 identify fiducial coordinates for each fiducial captured in the data acquired by the imaging 

apparatus. In accordance with some embodiments, the computer readable medium may 
further cause an apparatus executing the instructions selectively to iterate identifying 
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coordinates for each fiducial and calculating an absolute location of identified acquired 
image feature centers. 

The computer readable medium may further cause an apparatus executing the 
instructions to utilize a linear least squares operation or similar statistical fitting function. 
5 Additionally, some disclosed embodiments of a computer readable medium cause an 

apparatus executing the instructions to assume that a rotation of the imaging apparatus 
relative to a fiducial grid is negligible. 

The foregoing and other aspects of the disclosed embodiments will be more fully 
understood through examination of the following detailed description thereof in conjunction 
1 0 with the drawing figures. 

BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 is a simplified diagram illustrating raw image data acquired by an imaging 
apparatus and representing a top view of a precision Cartesian grid of fiducials printed on a 
fiducial plate. 

15 FIG. 2 is a simplified diagram depicting an exemplary set of fiducial locations 

derived from raw image data. 

FIG. 3 is a simplified diagram illustrating image data processed in accordance with 
one embodiment of a fiducial fitting technique. 

DETAILED DESCRIPTION 

20 By way of background, it is noted that if the number of fiducials in a given field of 

view {i.e., area of a fiducial plate imaged by an imaging apparatus) is large, fiducial location 
measurement noise may be reduced by optimally fitting the acquired image data to a fiducial 
grid model, and by using the resulting identified model significantly to improve 
measurement accuracy relative to interpolation from a single fiducial or from a small set of 

25 fiducials. One exemplary approach described herein generally involves fitting a fiducial grid 

model to measured (i.e., "acquired") data, establishing a conversion from camera (i.e., 
"acquired") coordinates to ideal fiducial coordinates, and calculating the absolute location of 
identified camera image feature centers in fiducial plate coordinates. 

It will be appreciated that the term "camera" in this context, and as used generally 

30 herein, is intended to encompass various imaging apparatus including, but not limited to, 

conventional optical cameras, digital cameras which may be embodied in or comprise 
charge-coupled device (CCD) or complementary metal-oxide semiconductor (CMOS) 
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device hardware and attendant electronics, and other optical or imaging hardware. These 
devices may comprise, or be implemented in conjunction with, various optical components 
such as lenses, mirrors, reflective or refractive grates, and the like, which may be configured 
and generally operative to achieve desired focal lengths, for example, or other operational 
5 characteristics. 

In accordance with one aspect of the present invention, fiducial movement may be 
tracked and a global coordinate reference may be maintained as the camera or imaging 
apparatus is translated from one location to another across a plane parallel to that of the 
fiducial plate. In that regard, and considering stage movement errors inherent in many 

10 mechanical or electromechanical systems, an algorithm such as those set forth in more detail 

below may rely upon stage error between discrete moves of less than or equal to half the 
center-to-center fiducial spacing (as measured on the fiducial plate). Mechanical stage 
movement errors larger than this may result in position measurement errors that are integer 
multiples of the fiducial spacing. 

1 5 Fiducial Grid Model 

In a given frame of acquired image data (z.e., data acquired by an imaging device, or 
"camera," in a single imaging operation), the number of acquired fiducial locations may be 
represented by a variable, n f . The x and y locations of the k* fiducial center, in camera pixel 
coordinates, may then be represented by (Xc pfk , y cpfk ) where k = 1, 2, . . . , n f . A particular 

20 fiducial, k, may be identified by its column, i pflc , and row, j pfk , relative to the origin of frame 

F at point S. Each coordinate in this local fiducial space reference frame (i pfk , j pflc ) may be 
mapped to a corresponding location in the camera space (x^ , y cp fk)- One exemplary 
approach for mapping local frame fiducial coordinates (i p£k , j pflc ) to camera coordinates (Xc k , 
y C k) may employ a polynomial fit in terms of fiducial coordinates as set forth below. 

25 Assuming a third order fit, the model may be expressed as: 

V : + 2 ^^+fX4/^ ..." 

The third order form of the foregoing model may be sufficient to capture or 
otherwise to quantify the following effects: 1) independent scale factors in the x and y 
directions (these scale factors may be due to a number of sources such as magnification and 
30 pixel size variation, for example, among other factors); 2) rotations about the z axis (optical 
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axis); 3) orthogonality errors in the camera pixel arrangement; and 4) keystone distortion 
caused by skewed viewing angle. 

The exemplary model may also adapt to or otherwise effectively account for other 
sources of image distortion, but may not capture these other effects exactly. If necessary or 
5 desired, fitting accuracy may be improved by selectively increasing the order of the 

polynomial fit. Using Equations (1) and (2), for example, it is possible to map coordinates in 
fiducial space to coordinates in camera pixel space, and vice-versa. The reverse mapping 
operation may require solving two non-linear equations in two unknowns, as is set forth in 
more detail below. The coordinates in fiducial space (i p , j p ) may be integer valued 
10 corresponding to actual fiducial locations ((i pf , j pf ) (Xc pf , y cpf )), or may be real valued 

corresponding to general camera pixel locations ((i p , j p ) — ► (xc P , y cp )). 

Fiducial Grid Model Fitting 
Fitting the measured camera frame fiducial locations (Xc P fk , y cp fk) to the fiducial 
model of Equations (1) and (2) may initially involve identifying the fiducial coordinates 
15 (integer row and column locations (i pflc , j P fc)) of all fiducials in the acquired image data 

frame. Since the grid of fiducials may have voids, for example, due to missing or occluded 
fiducials, a fully populated grid of fiducial coordinates (e.g., a full fiducial array) need not be 
assumed. One way to identify the fiducial coordinates of the measured fiducials is to use a 
simplified version of Equations (1) and (2) that includes only linear terms: 
x cpfk - x 0 = Zii Pfi + Zzj Pfk (3) 

It may be convenient to set quantities xo and y 0 to the pixel coordinates of an 
identified fiducial near the center of the field of view. For a known fiducial pitch (A^ , 
Ayfid), estimated camera magnification (M), and camera pixel dimensions (w pix , h pix ), inter- 
fiducial spacing in camera pixel coordinates may be estimated. Given an estimate of total 
25 camera rotation relative to the fiducial grid (9 t ), the parameters z\ 9 z 3 , Z4, and z 2 may be set as 

follows: 
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2 1 

z 2 = A y<1Affl cos(0/) 



(5) 

<6> 

(7) 
<8) 

(10) 



With sufficiently accurate alignment characteristics enabled by present mechanical 
systems, control components, and electronics, it may be reasonable to assume 8 t = 0. It will 
be appreciated by those of skill in the art that, given a reasonable estimate of xo, y 0 , Z\ 9 z 3 , Z4 
and z 2 , Equations (3) and (4) may be solved for fiducial coordinates i pfk and j pfk . In that 
regard, A init and y init may be defined as follows: 



Zl z 3 

•it 22 



lc Pfk 



Then, from Equations (3) through (12): 



(11) 

(12): 



(13) 
(14) 
(15) 



1 0 where the "round" function rounds the argument to the nearest integer. 

Now with an estimate of the locations of the fiducials in fiducial coordinates (the 
row and column number of the measured fiducials) given by Equations (14) and (15), it is 
possible to return to the third order model and to solve for the unknown parameters using, 
for example, a linear least squares method. Equations (1) and (2) may be recast into matrix 

15 form via: 
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A 0l A h i4 2 , A 3 , 
A 0i A^ Az 2 A h 

AV A \ A l> A \_ 



(16) 



y = Ap 



P 



(17) 



where 



40, -i 






.° J>* 0 *p/ij 




'p/i 'p/i/p/i /p/i 0 
_P 0 0 £ 


A 3j = 


'p/i 'p/^p/i Hfiiffi 
0 0 0 



■ 0 :- : .:vg-;; 
0 0 



0 



(20) 
(21) 
(22) 



Solving Equation (18) employing linear least squares produces the best fit grid 
parameters, p, in accordance with Equation (23) as set forth below: 



(23) 



Depending upon the accuracy of the estimates for z u z 3 , Z4, and z 2 used to solve 
Equation (13) for the coordinates of fiducials in fiducial space, it may be necessary to iterate 
between Equations (13) and (23) to arrive at a stable solution for p. In practice, this iterative 
process has been determined to converge very rapidly; two iterations may typically be 
sufficient for suitable convergence. 

FIG. 2 is a simplified diagram depicting an exemplary set of fiducial locations 
derived from raw image data. In that regard, the fiducial locations illustrated in the FIG. 2 
image are derived from the raw image data illustrated in FIG 1. The result of applying the 
exemplary fiducial fitting technique set forth herein is illustrated in FIG. 3. Specifically, 
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FIG. 3 is a simplified diagram illustrating image data processed in accordance with one 
embodiment of a fiducial fitting technique. 

In the FIG. 3 illustration, fiducials are depicted as dark, filled dots, while each 
identified fiducial is indicated by the presence of an unfilled circle described around the 
5 respective dark dot. The network of intersecting lines in FIG. 3 represents lines of constant 

x and y in the fiducial coordinate system. Note that in the image pixel coordinate system, 
these "lines" appear distorted, and show significant keystone^arrel effects. 

Fiducial Grid Model Inversion 
As noted briefly above, inverting Equation (1) and (2) to solve for i p and j p 
10 corresponding to a desired camera pixel coordinate (Xc P = x^s , y cp = y cp des) may generally 

involve solving two non-linear equations in two unknowns. In one exemplary embodiment, 
the equations to be solved are set forth below: 

x cp « * 0 ±zii p + zsjp +z s i p j P + ir% +ziHp 4- Z\#%+Z\3i p jp + Zwf p (24) 
: iy^p- yo + Z41V+ z 2?> t z&l f BWp + z ^iv + z ^p .+ ^pjv + 2 i#Vp + 218 /p C 25 ) 

Typically the coefficients of the non-linear terms are very small, and an initial 
15 estimate of the linear coefficients may be calculated, for example, by assuming that these 

non-linear coefficients are zero. Under this assumption, 

•: l/tinlT ^linPlm. ■ y. j - ,.\: ' W) ; ,;: ,; 

and the linearized approximation, p^, may be solved through a simple matrix inversion 

|lin ^ihlyiim ; 



20 This initial linear estimate for (i p , j p ) may be used as a starting value for an iterative 

solution of non-linear Equations (24) and (25). 

Conjugate Gradient Search 
Alternatively, a non-linear least squares solution may be employed. The selected 
cost function to be minimized in this embodiment may be the square of the Euclidean 
25 distance between the desired camera coordinate (x^es , y cp des) and the model predicted 

camera coordinate (Xc P , y cp ). This cost function may be written as 
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The gradient of this cost function with respect to the fiducial coordinates i and j may be 
expressed as follows: 

dx 

= zi + 2i p z 5 + j v U + 3zi iij + 2zi 2 i p j P + Z13/J (30) 

^ = Z4 H- 2i„z 8 + /pZ 9 + 3zi 5 iJ + 2zi fi iy p 4- z 17 /p (31) 

^jp -z 3 + ipZ 6 + 2/ p z 7 Wz\ppz<& p } p + 3zw/J (32) 

^ = z 2 H-«pZ 9 +2/pZi 0 + z, 6 iJ + 2z, 7 i p /p4-3^ , (33) 

^ = 2(x C p-x^)^ + 2(y,^^)^ ; (34) 

Given the cost function and analytic gradients set forth in Equations (29) through 
(35), Equations (24) and (25) may be solved using any of a number of suitable conjugate 
gradient search algorithms. Given the typically good estimate provided by the approximate 
linear solution, the conjugate gradient search converges very quickly in practice (typically 
four iterations or fewer are sufficient for convergence). 

Iterative Cubic Equation Solution 

In another alternative embodiment, Equations (24) and (25) may be solved 
iteratively as cubic equations in i and j, respectively, substantially as set forth below. 
Rearranging terms: 

+ (zs + mip) § -f (zi + zsjp + zi3$) ip H- (xq ~ x cp + z*}p ±Z7}\+ ztift): f 0 ( 36 > 

2 Wp+ ( 2 10+2l7/ P )/p+ (z 2 + Z 9 i p + Z ls ify -0. (37) 

Defining coefficients, Equations (36) and (37) become 

aul + b\i 2 p + cv p + d<i -0 (38) 
«ij 3 p + b2}\ + c 2 i p + d 2 - 0 (39) 

where 
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d t 4 x Q — Xcp +2 3 / p +27/p +Z14/J 
&2 = 210 +2l7^ 



(40) 
(41) 
(42) 
(43) 
(44) 
(45) 
(46) 
(47) 



Given the linear solution pi in from Equation (28) as a starting point, values may be 
assigned to a b bi, and Ci for an assumed j p . Equation (38) may then be solved for i p , and the 
root nearest i pHn may be selected. Some methods may take this new value for i p and assign 
appropriate values to a 2 , t>2, and c 2 . Equation (39) may then be solved for j p and the root 
nearest j p i in may be selected. The foregoing process may result in an improved solution 
estimate (i p , j p ). The process may be iterated until the solution converges to a specified or 
predetermined tolerance. In practice, only three iterations are typically required for 
convergence to a point where the distance from the current estimate (i p , j p ) to the previous 
estimate is less than 1 x 10" 6 . 

It will be appreciated that the foregoing functionality may be achieved, and that the 
equations set forth above may be solved or approximated, by suitable data processing 
hardware and software components generally known in the art and appropriately configured 
and programmed. Typical image acquisition systems employ such data processing hardware 
and attendant software, either of which may readily be updated, augmented, modified, or 
otherwise reprogrammed with computer executable instructions operative to cause the data 
processing hardware to compute solutions or approximations to the equations outlined 
above. Additionally, it will be apparent to those of skill in the art that the foregoing 
embodiments may be susceptible of various modifications within the scope and 
contemplation of the present disclosure. By way of example, the exemplary embodiments 
are not intended to be limited to any particular polynomial functions, for instance, or 
conjugate gradient search algorithms. 

Aspects of the present invention have been illustrated and described in detail with 
reference to particular embodiments by way of example only, and not by way of limitation. 
It will be appreciated that various modifications and alterations may be made to the 
exemplary embodiments without departing from the scope and contemplation of the present 



11 



disclosure. It is intended, therefore, that the invention be considered as limited only 
scope of the appended claims 



12 



